Data integration may be described as extracting data from a source, transforming the data, and loading the data to a target. That is, data integration is Extract, Transform, Load (ETL) processing. Data integration processing engines may be scalable and capable of processing large volumes of data in complex data integration projects. It is common for multiple users (e.g., customers) and projects to share a single data integration processing engine that is responsible for handling all of the data integration processing for those multiple users. This high volume, highly concurrent processing may be resource intensive, and users try to balance the availability of system resources with the need to process large volumes of data efficiently and concurrently.
Workload management capabilities may be available at Operating System (OS) or lower levels. Workload management operates at a level that is removed from the data integration environment.
Some users use a multi-node grid so that they can utilize a grid resource manager to better control system resources. The grid resource manager deploys data integration projects on a grid.
Some users use Symmetric Multiprocessing (SMP), which are multi-core, hyper-threaded systems for data integration.
Some users rely on workload schedulers to control workloads. This requires coordination between different groups or the same group running different projects.